Microlensing for Real-Time Sensing of Stray Light

ABSTRACT

Example embodiments relate to microlensing for real-time sensing of stray light. An example device includes an image sensor that includes a plurality of light-sensitive pixels. The device also includes a first lens positioned over a first subset of light-sensitive pixels selected from the plurality of light-sensitive pixels. Further, the device includes a controller. The controller is configured to determine a first angle of incidence of a first light signal detected by the first subset of light-sensitive pixels. The controller is also configured to, based on the first determined angle of incidence, determine an amount of stray light incident on the image sensor.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority to Provisional PatentApplication No. 62/953,721, filed with the U.S. Patent and TrademarkOffice on Dec. 26, 2019, the contents of which are hereby incorporatedby reference.

BACKGROUND

Unless otherwise indicated herein, the materials described in thissection are not prior art to the claims in this application and are notadmitted to be prior art by inclusion in this section.

Cameras and image sensors are devices used to capture images of a scene.Some cameras (e.g., film cameras) chemically capture an image on film.Other cameras (e.g., digital cameras) electrically capture image data(e.g., using a charge-coupled device (CCD) or complementarymetal-oxide-semiconductor (CMOS) sensors). Images captured by camerascan be analyzed to determine their contents. For example, a processormay execute a machine-learning algorithm in order to identify objects ina scene based on a library of previously classified objects thatincludes objects' shapes, colors, sizes, etc. (e.g., such amachine-learning algorithm can be applied in computer vision in roboticsor other applications).

Cameras can have a variety of features that can distinguish one camerafrom another. For example, cameras and/or images captured by cameras maybe identified by values such as aperture size, f-number, exposure time,shutter speed, depth of field, focal length, International Organizationfor Standardization (ISO) sensitivity (or gain), pixel size, sensorresolution, exposure distance, etc. These features may be based on thelens, the image sensor, and/or additional facets of the camera. Further,these features may also be adjustable within a single camera (e.g., theaperture of a lens on a camera can be adjusted between photographs).

Further, as a result of imperfections of optics within a camera,aberrations within captured images can be generated. For example,misalignment, non-unity transmittance, internal reflections, debris,etc. may cause light from a scene to be directed to unintended/improperregions of an image sensor. Such stray light may appear as veiling glareor lens flare within a captured image.

SUMMARY

This disclosure relates to detecting (and, in some embodiments,compensating for) stray light incident on the image sensor of a camerausing microlenses. The microlenses may be positioned over a set oflight-sensitive pixels on the image sensor. Based on the signalintensity detected by each of the light-sensitive pixels underlying themicrolens, an angle of incidence of an incoming light signal may bedetermined. Based on the determined angle of incidence, the incominglight signal can be identified as stray light (e.g., veiling glareand/or a ghost image) or an actual image signal. In some embodiments,corrections in hardware or software may be employed to compensate forthe stray light incident on the image sensor and/or to reduce the amountof stray incident on the image sensor based on the identified straylight.

In one aspect, a device is provided. The device includes an image sensorthat includes a plurality of light-sensitive pixels. The device alsoincludes a first lens positioned over a first subset of light-sensitivepixels selected from the plurality of light-sensitive pixels. Further,the device includes a controller. The controller is configured todetermine a first angle of incidence of a first light signal detected bythe first subset of light-sensitive pixels. The controller is alsoconfigured to, based on the first determined angle of incidence,determine an amount of stray light incident on the image sensor.

In another aspect, a method is provided. The method includes receiving,at a lens positioned over a first subset of light-sensitive pixelsselected from a plurality of light-sensitive pixels that are part of animage sensor, a first light signal. The method also includes directing,using the lens, the first light signal toward the first subset oflight-sensitive pixels. Further, the method includes detecting, by oneor more light-sensitive pixels of the first subset, the first lightsignal. In addition, the method includes determining a first angle ofincidence of the detected first light signal. Still further, the methodincludes determining, based on the first determined angle of incidence,an amount of stray light incident on the image sensor.

In an additional aspect, a device is provided. The device includes animage sensor that includes a plurality of light-sensitive pixels. Thedevice also includes a plurality of subsets of light-sensitive pixelsselected from the plurality of light-sensitive pixels positioned alongan entire periphery of the image sensor. Further, the device includes aplurality of lenses. Each lens is positioned over a corresponding subsetof light-sensitive pixels. In addition, the device includes acontroller. The controller is configured to determine the angle ofincidence of light detected by each subset of light-sensitive pixels.The controller is also configured to, based on the determined angles ofincidence, determine a stray-light map across the image sensor.

These as well as other aspects, advantages, and alternatives will becomeapparent to those of ordinary skill in the art by reading the followingdetailed description, with reference, where appropriate, to theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram illustrating a vehicle, accordingto example embodiments.

FIG. 2A is an illustration of a physical configuration of a vehicle,according to example embodiments.

FIG. 2B is an illustration of a physical configuration of a vehicle,according to example embodiments.

FIG. 2C is an illustration of a physical configuration of a vehicle,according to example embodiments.

FIG. 2D is an illustration of a physical configuration of a vehicle,according to example embodiments.

FIG. 2E is an illustration of a physical configuration of a vehicle,according to example embodiments.

FIG. 3 is a conceptual illustration of wireless communication betweenvarious computing systems related to an autonomous vehicle, according toexample embodiments.

FIG. 4A is an illustration of a camera, according to exampleembodiments.

FIG. 4B is an illustration of a mechanism that gives rise to veilingglare within an optical detector system, according to exampleembodiments.

FIG. 4C is an illustration of a mechanism that gives rise to veilingglare within an optical detector system, according to exampleembodiments.

FIG. 4D is an illustration of a mechanism that gives rise to veilingglare within an optical detector system, according to exampleembodiments.

FIG. 5 is an image of a scene that includes a bright object, accordingto example embodiments.

FIG. 6A is an illustration of an image sensor.

FIG. 6B is an illustration of an image sensor.

FIG. 7A is an illustration of an image sensor, according to exampleembodiments.

FIG. 7B is an illustration of an image sensor, according to exampleembodiments.

FIG. 7C is an illustration of a subset of light-sensitive pixels with anassociated lens, according to example embodiments.

FIG. 7D is an illustration of a subset of light-sensitive pixels with anassociated lens, according to example embodiments.

FIG. 7E is an illustration of a subset of light-sensitive pixels with anassociated lens, according to example embodiments.

FIG. 7F is an illustration of a subset of light-sensitive pixels with anassociated lens, according to example embodiments.

FIG. 8A is an illustration of a generated stray-light map, according toexample embodiments.

FIG. 8B is an illustration of a generated stray-light map, according toexample embodiments.

FIG. 9A is an illustration of an image sensor, according to exampleembodiments.

FIG. 9B is an illustration of an image sensor, according to exampleembodiments.

FIG. 10A is an illustration of an image sensor, according to exampleembodiments.

FIG. 10B is an illustration of an image sensor, according to exampleembodiments.

FIG. 11 is an illustration of a method, according to exampleembodiments.

DETAILED DESCRIPTION

Example methods and systems are contemplated herein. Any exampleembodiment or feature described herein is not necessarily to beconstrued as preferred or advantageous over other embodiments orfeatures. The example embodiments described herein are not meant to belimiting. It will be readily understood that certain aspects of thedisclosed systems and methods can be arranged and combined in a widevariety of different configurations, all of which are contemplatedherein.

Furthermore, the particular arrangements shown in the figures should notbe viewed as limiting. It should be understood that other embodimentsmight include more or less of each element shown in a given figure.Further, some of the illustrated elements may be combined or omitted.Yet further, an example embodiment may include elements that are notillustrated in the figures.

I. OVERVIEW

As a result of internal reflections within a camera (e.g., off of opticsin the camera, off of mounting components in the camera, off ofelectronics in the camera, due to imperfections and/or misalignments ofcomponents within the camera, etc.), aberrations within captured images(e.g., payload images used for object identification) can be generated.As such, light from a scene may be directed to unintended or improperregions of an image sensor. For example, light entering a camera from apart of a scene that would normally be outside of a field of view of thecamera may be internally reflected by a lens housing, redirected to theimage sensor of the camera, and detected. Such extraneous light (i.e.,stray light) may appear as veiling glare or lens flare within a capturedimage. Additionally or alternatively, stray light may appear as one ormore ghost images within the captured image.

To enhance the quality of captured images (e.g., to improve objectidentification in computer-vision applications, such as autonomousvehicle applications), it may be desirable to identify and/or removeveiling glare, lens flare, ghost images, and/or other aberrations causedby stray light from captured images. Techniques described herein presentways of identifying and/or removing stray light captured by a camera.

Many cameras include a micro-lens array that overlays the image sensor.The micro-lens array may have a plurality of micro-lenses eachcorresponding to a light-sensitive pixel within the image sensor (e.g.,to direct more of the light from the scene to the light-sensitiveportion of the light-sensitive pixel). One technique described hereinfor identifying stray light includes positioning a slightly largermicro-lens (e.g., a micro-lens that overlays four pixels, nine pixels,or sixteen pixels) over a subset of light-sensitive pixels within theimage sensor. Such a larger micro-lens may be installed in addition toor instead of the portion of the micro-lens array that would otherwisebe present (e.g., a larger micro-lens over a 3×3 square of pixels mayreplace the nine smaller micro-lenses that would otherwise be present).This larger micro-lens may direct incoming light signals to one or moreof the light-sensitive pixels within the subset (e.g., may direct anincoming light signal to one or more of the nine pixels in a 3×3arrangement). Based on which of the underlying light-sensitive pixelsreceive light, and in what relative intensities, an angle of incidencefor the incoming light signal may be determined. Such a determinationmay be made by a controller of the camera, for example. Alternatively,in some embodiments, such a determination may be made by a separatecomputing device (e.g., a server receiving a copy of the captured image,a laptop computing device receiving a copy of the captured image, oranother related processor, such as a processor of a mobile computingdevice executing instructions stored within a memory of the mobilecomputing device to assess images captured by the camera).

In example embodiments, light coming from the target field of view ofthe scene is incident upon the subset of light-sensitive pixels at aperpendicular angle or near-perpendicular angle. Hence, there is athreshold angle of incidence above which it can be determined that thelight signal being detected by the subset of light-sensitive pixelsentered from some spurious direction and, therefore, corresponds to astray light signal. Thus, by comparing the determined angle of incidenceof the incoming light signal to a threshold angle of incidence, it canbe determined whether the detected light signal is a stray light signal.If the light signal is a stray light signal, the stray light signal canbe eliminated in post-processing (e.g., by selectively decreasingintensity in corresponding regions of stray light within payloadimages). Additionally or alternatively, the intensity of the stray lightsignal can be used to determine a true-black optical level for thecamera. Still further, if stray light is determined to be present in acaptured image, the pose of the camera can be adjusted, the lens of thecamera can be adjusted, and/or one or more filters may be applied to thecamera before additional images are captured.

These subsets of light-sensitive pixels used for stray light detectionmay be dedicated subsets of light-sensitive pixels, in some embodiments.In other words, these subsets of light-sensitive pixels may be usedsolely for stray light detection. As such, so as not to interfere withthe bulk of captured payload images, these subsets of light-sensitivepixels may be positioned around a periphery of the image sensor (e.g.,in a corner of the image sensor or along an edge of the image sensor).Alternatively or additionally, light-sensitive pixels used for straylight detection may be located in the bulk of the image sensor, such asnear a center of the image sensor. Because these stray-light detectionsubsets can be relatively small (e.g., four pixels or nine pixels), theycould be positioned in the bulk of the image sensor without causing toomuch disruption to a captured payload image. The image locationscorresponding to these stray-light detection subsets in the bulk of theimage sensor could be accounted for in post-processing, as well (e.g.,based on the non-stray-light detecting light-sensitive pixels near thestray-light detecting light-sensitive pixels, the light from the sceneincident on the stray-light detecting light-sensitive pixels may beestimated and integrated into a payload image).

In some embodiments, in order to more accurately identify the region(s)of the image sensor that might be susceptible to stray light, the entireperiphery of the image sensor may include these stray-light determiningsubsets. Based on measurements of stray light along the entire peripheryof the image sensor, the stray light within the bulk of the image sensorcan be projected using interpolation (e.g., by generating a log-encodedor a gamma-encoded histogram that can be stored in a look-up table forlater use).

Many cameras include multiple color channels (e.g., red, green, blue).Hence, because some of the light-sensitive pixels within the imagesensor may be used to detect different colors (e.g., red, green, blue),there may be different stray-light determining subsets dedicated to eachcolor (e.g., some subsets dedicated to red stray-light determinations,some subsets dedicated to green stray-light determinations, and somesubsets dedicated to blue stray-light determinations). Each of thelight-sensitive pixels in the underlying color-specific subset maycorrespond to the same color (e.g., nine red pixels under a singlemicro-lens, nine green pixels under a single micro-lens, or nine bluepixels under a single micro-lens). Using the information from thesedifferent color-specific subsets, multiple stray light maps could begenerated for the image sensor (e.g., a red stray-light map, a greenstray-light map, and a blue stray-light map). As such, these dedicatedstray-light subsets may improve demosaicing, as well.

II. EXAMPLE SYSTEMS

The following description and accompanying drawings will elucidatefeatures of various example embodiments. The embodiments provided are byway of example, and are not intended to be limiting. As such, thedimensions of the drawings are not necessarily to scale.

Example systems within the scope of the present disclosure will now bedescribed in greater detail. An example system may be implemented in ormay take the form of an automobile. However, an example system may alsobe implemented in or take the form of other vehicles, such as cars,trucks, motorcycles, buses, boats, airplanes, helicopters, lawn mowers,earth movers, boats, snowmobiles, aircraft, recreational vehicles,amusement park vehicles, farm equipment, construction equipment, trams,golf carts, trains, trolleys, and robot devices. Other vehicles arepossible as well. Further, in some embodiments, example systems mightnot include a vehicle.

Referring now to the figures, FIG. 1 is a functional block diagramillustrating example vehicle 100, which may be configured to operatefully or partially in an autonomous mode. More specifically, vehicle 100may operate in an autonomous mode without human interaction throughreceiving control instructions from a computing system. As part ofoperating in the autonomous mode, vehicle 100 may use sensors to detectand possibly identify objects of the surrounding environment to enablesafe navigation. In some embodiments, vehicle 100 may also includesubsystems that enable a driver to control operations of vehicle 100.

As shown in FIG. 1, vehicle 100 may include various subsystems, such aspropulsion system 102, sensor system 104, control system 106, one ormore peripherals 108, power supply 110, computer system 112 (could alsobe referred to as a computing system), data storage 114, and userinterface 116. In other examples, vehicle 100 may include more or fewersubsystems, which can each include multiple elements. The subsystems andcomponents of vehicle 100 may be interconnected in various ways. Inaddition, functions of vehicle 100 described herein can be divided intoadditional functional or physical components, or combined into fewerfunctional or physical components within embodiments. For instance, thecontrol system 106 and the computer system 112 may be combined into asingle system that operates the vehicle 100 in accordance with variousoperations.

Propulsion system 102 may include one or more components operable toprovide powered motion for vehicle 100 and can include an engine/motor118, an energy source 119, a transmission 120, and wheels/tires 121,among other possible components. For example, engine/motor 118 may beconfigured to convert energy source 119 into mechanical energy and cancorrespond to one or a combination of an internal combustion engine, anelectric motor, steam engine, or Stirling engine, among other possibleoptions. For instance, in some embodiments, propulsion system 102 mayinclude multiple types of engines and/or motors, such as a gasolineengine and an electric motor.

Energy source 119 represents a source of energy that may, in full or inpart, power one or more systems of vehicle 100 (e.g., engine/motor 118).For instance, energy source 119 can correspond to gasoline, diesel,other petroleum-based fuels, propane, other compressed gas-based fuels,ethanol, solar panels, batteries, and/or other sources of electricalpower. In some embodiments, energy source 119 may include a combinationof fuel tanks, batteries, capacitors, and/or flywheels.

Transmission 120 may transmit mechanical power from engine/motor 118 towheels/tires 121 and/or other possible systems of vehicle 100. As such,transmission 120 may include a gearbox, a clutch, a differential, and adrive shaft, among other possible components. A drive shaft may includeaxles that connect to one or more wheels/tires 121.

Wheels/tires 121 of vehicle 100 may have various configurations withinexample embodiments. For instance, vehicle 100 may exist in a unicycle,bicycle/motorcycle, tricycle, or car/truck four-wheel format, amongother possible configurations. As such, wheels/tires 121 may connect tovehicle 100 in various ways and can exist in different materials, suchas metal and rubber.

Sensor system 104 can include various types of sensors, such as GlobalPositioning System (GPS) 122, inertial measurement unit (IMU) 124, radar126, laser rangefinder/lidar 128, camera 130, steering sensor 123, andthrottle/brake sensor 125, among other possible sensors. In someembodiments, sensor system 104 may also include sensors configured tomonitor internal systems of the vehicle 100 (e.g., 02 monitor, fuelgauge, engine oil temperature, brake wear).

GPS 122 may include a transceiver operable to provide informationregarding the position of vehicle 100 with respect to the Earth. IMU 124may have a configuration that uses one or more accelerometers and/orgyroscopes and may sense position and orientation changes of vehicle 100based on inertial acceleration. For example, IMU 124 may detect a pitchand yaw of the vehicle 100 while vehicle 100 is stationary or in motion.

Radar 126 may represent one or more systems configured to use radiosignals to sense objects, including the speed and heading of theobjects, within the local environment of vehicle 100. As such, radar 126may include antennas configured to transmit and receive radio signals.In some embodiments, radar 126 may correspond to a mountable radarsystem configured to obtain measurements of the surrounding environmentof vehicle 100.

Laser rangefinder/lidar 128 may include one or more laser sources, alaser scanner, and one or more detectors, among other system components,and may operate in a coherent mode (e.g., using heterodyne detection) orin an incoherent detection mode. In some embodiments, the one or moredetectors of the laser rangefinder/lidar 128 may include one or morephotodetectors. Such photodetectors may be especially sensitivedetectors (e.g., avalanche photodiodes (APDs)). In some examples, suchphotodetectors may even be capable of detecting single photons (e.g.,single-photon avalanche diodes (SPADs)). Further, such photodetectorscan be arranged (e.g., through an electrical connection in series) intoan array (e.g., as in a silicon photomultiplier (SiPM)).

Camera 130 may include one or more devices (e.g., still camera or videocamera) configured to capture images of the environment of vehicle 100.

Steering sensor 123 may sense a steering angle of vehicle 100, which mayinvolve measuring an angle of the steering wheel or measuring anelectrical signal representative of the angle of the steering wheel. Insome embodiments, steering sensor 123 may measure an angle of the wheelsof the vehicle 100, such as detecting an angle of the wheels withrespect to a forward axis of the vehicle 100. Steering sensor 123 mayalso be configured to measure a combination (or a subset) of the angleof the steering wheel, electrical signal representing the angle of thesteering wheel, and the angle of the wheels of vehicle 100.

Throttle/brake sensor 125 may detect the position of either the throttleposition or brake position of vehicle 100. For instance, throttle/brakesensor 125 may measure the angle of both the gas pedal (throttle) andbrake pedal or may measure an electrical signal that could represent,for instance, an angle of a gas pedal (throttle) and/or an angle of abrake pedal. Throttle/brake sensor 125 may also measure an angle of athrottle body of vehicle 100, which may include part of the physicalmechanism that provides modulation of energy source 119 to engine/motor118 (e.g., a butterfly valve or carburetor). Additionally,throttle/brake sensor 125 may measure a pressure of one or more brakepads on a rotor of vehicle 100 or a combination (or a subset) of theangle of the gas pedal (throttle) and brake pedal, electrical signalrepresenting the angle of the gas pedal (throttle) and brake pedal, theangle of the throttle body, and the pressure that at least one brake padis applying to a rotor of vehicle 100. In other embodiments,throttle/brake sensor 125 may be configured to measure a pressureapplied to a pedal of the vehicle, such as a throttle or brake pedal.

Control system 106 may include components configured to assist innavigating vehicle 100, such as steering unit 132, throttle 134, brakeunit 136, sensor fusion algorithm 138, computer vision system 140,navigation/pathing system 142, and obstacle avoidance system 144. Morespecifically, steering unit 132 may be operable to adjust the heading ofvehicle 100, and throttle 134 may control the operating speed ofengine/motor 118 to control the acceleration of vehicle 100. Brake unit136 may decelerate vehicle 100, which may involve using friction todecelerate wheels/tires 121. In some embodiments, brake unit 136 mayconvert kinetic energy of wheels/tires 121 to electric current forsubsequent use by a system or systems of vehicle 100.

Sensor fusion algorithm 138 may include a Kalman filter, Bayesiannetwork, or other algorithms that can process data from sensor system104. In some embodiments, sensor fusion algorithm 138 may provideassessments based on incoming sensor data, such as evaluations ofindividual objects and/or features, evaluations of a particularsituation, and/or evaluations of potential impacts within a givensituation.

Computer vision system 140 may include hardware and software operable toprocess and analyze images in an effort to determine objects,environmental objects (e.g., traffic lights, roadway boundaries, etc.),and obstacles. As such, computer vision system 140 may use objectrecognition, Structure From Motion (SFM), video tracking, and otheralgorithms used in computer vision, for instance, to recognize objects,map an environment, track objects, estimate the speed of objects, etc.

Navigation/pathing system 142 may determine a driving path for vehicle100, which may involve dynamically adjusting navigation duringoperation. As such, navigation/pathing system 142 may use data fromsensor fusion algorithm 138, GPS 122, and maps, among other sources tonavigate vehicle 100. Obstacle avoidance system 144 may evaluatepotential obstacles based on sensor data and cause systems of vehicle100 to avoid or otherwise negotiate the potential obstacles.

As shown in FIG. 1, vehicle 100 may also include peripherals 108, suchas wireless communication system 146, touchscreen 148, microphone 150,and/or speaker 152. Peripherals 108 may provide controls or otherelements for a user to interact with user interface 116. For example,touchscreen 148 may provide information to users of vehicle 100. Userinterface 116 may also accept input from the user via touchscreen 148.Peripherals 108 may also enable vehicle 100 to communicate with devices,such as other vehicle devices.

Wireless communication system 146 may wirelessly communicate with one ormore devices directly or via a communication network. For example,wireless communication system 146 could use 3G cellular communication,such as code-division multiple access (CDMA), evolution-data optimized(EVDO), global system for mobile communications (GSM)/general packetradio service (GPRS), or 4G cellular communication, such as worldwideinteroperability for microwave access (WiMAX) or long-term evolution(LTE). Alternatively, wireless communication system 146 may communicatewith a wireless local area network (WLAN) using WiFi or other possibleconnections. Wireless communication system 146 may also communicatedirectly with a device using an infrared link, Bluetooth, or ZigBee, forexample. Other wireless protocols, such as various vehicularcommunication systems, are possible within the context of thedisclosure. For example, wireless communication system 146 may includeone or more dedicated short-range communications (DSRC) devices thatcould include public and/or private data communications between vehiclesand/or roadside stations.

Vehicle 100 may include power supply 110 for powering components. Powersupply 110 may include a rechargeable lithium-ion or lead-acid batteryin some embodiments. For instance, power supply 110 may include one ormore batteries configured to provide electrical power. Vehicle 100 mayalso use other types of power supplies. In an example embodiment, powersupply 110 and energy source 119 may be integrated into a single energysource.

Vehicle 100 may also include computer system 112 to perform operations,such as operations described therein. As such, computer system 112 mayinclude at least one processor 113 (which could include at least onemicroprocessor) operable to execute instructions 115 stored in anon-transitory, computer-readable medium, such as data storage 114. Insome embodiments, computer system 112 may represent a plurality ofcomputing devices that may serve to control individual components orsubsystems of vehicle 100 in a distributed fashion.

In some embodiments, data storage 114 may contain instructions 115(e.g., program logic) executable by processor 113 to execute variousfunctions of vehicle 100, including those described above in connectionwith FIG. 1. Data storage 114 may contain additional instructions aswell, including instructions to transmit data to, receive data from,interact with, and/or control one or more of propulsion system 102,sensor system 104, control system 106, and peripherals 108.

In addition to instructions 115, data storage 114 may store data such asroadway maps, path information, among other information. Suchinformation may be used by vehicle 100 and computer system 112 duringthe operation of vehicle 100 in the autonomous, semi-autonomous, and/ormanual modes.

Vehicle 100 may include user interface 116 for providing information toor receiving input from a user of vehicle 100. User interface 116 maycontrol or enable control of content and/or the layout of interactiveimages that could be displayed on touchscreen 148. Further, userinterface 116 could include one or more input/output devices within theset of peripherals 108, such as wireless communication system 146,touchscreen 148, microphone 150, and speaker 152.

Computer system 112 may control the function of vehicle 100 based oninputs received from various subsystems (e.g., propulsion system 102,sensor system 104, and control system 106), as well as from userinterface 116. For example, computer system 112 may utilize input fromsensor system 104 in order to estimate the output produced by propulsionsystem 102 and control system 106. Depending upon the embodiment,computer system 112 could be operable to monitor many aspects of vehicle100 and its subsystems. In some embodiments, computer system 112 maydisable some or all functions of the vehicle 100 based on signalsreceived from sensor system 104.

The components of vehicle 100 could be configured to work in aninterconnected fashion with other components within or outside theirrespective systems. For instance, in an example embodiment, camera 130could capture a plurality of images that could represent informationabout a state of an environment of vehicle 100 operating in anautonomous mode. The state of the environment could include parametersof the road on which the vehicle is operating. For example, computervision system 140 may be able to recognize the slope (grade) or otherfeatures based on the plurality of images of a roadway. Additionally,the combination of GPS 122 and the features recognized by computervision system 140 may be used with map data stored in data storage 114to determine specific road parameters. Further, radar 126 may alsoprovide information about the surroundings of the vehicle.

In other words, a combination of various sensors (which could be termedinput-indication and output-indication sensors) and computer system 112could interact to provide an indication of an input provided to controla vehicle or an indication of the surroundings of a vehicle.

In some embodiments, computer system 112 may make a determination aboutvarious objects based on data that is provided by systems other than theradio system. For example, vehicle 100 may have lasers or other opticalsensors configured to sense objects in a field of view of the vehicle.Computer system 112 may use the outputs from the various sensors todetermine information about objects in a field of view of the vehicle,and may determine distance and direction information to the variousobjects. Computer system 112 may also determine whether objects aredesirable or undesirable based on the outputs from the various sensors.

Although FIG. 1 shows various components of vehicle 100 (i.e., wirelesscommunication system 146, computer system 112, data storage 114, anduser interface 116) as being integrated into the vehicle 100, one ormore of these components could be mounted or associated separately fromvehicle 100. For example, data storage 114 could, in part or in full,exist separate from vehicle 100. Thus, vehicle 100 could be provided inthe form of device elements that may be located separately or together.The device elements that make up vehicle 100 could be communicativelycoupled together in a wired and/or wireless fashion.

FIGS. 2A-2E shows an example vehicle 200 that can include some or all ofthe functions described in connection with vehicle 100 in reference toFIG. 1. Although vehicle 200 is illustrated in FIGS. 2A-2E as a van forillustrative purposes, the present disclosure is not so limited. Forinstance, the vehicle 200 can represent a truck, a car, a semi-trailertruck, a motorcycle, a golf cart, an off-road vehicle, a farm vehicle,etc.

The example vehicle 200 includes a sensor unit 202, a first lidar unit204, a second lidar unit 206, a first radar unit 208, a second radarunit 210, a first lidar/radar unit 212, a second lidar/radar unit 214,and two additional locations 216, 218 at which a radar unit, lidar unit,laser rangefinder unit, and/or other type of sensor or sensor(s) couldbe located on the vehicle 200. Each of the first lidar/radar unit 212and the second lidar/radar unit 214 can take the form of a lidar unit, aradar unit, or both.

Furthermore, the example vehicle 200 can include any of the componentsdescribed in connection with vehicle 100 of FIG. 1. The first and secondradar units 208, 210 and/or the first and second lidar units 204, 206can actively scan the surrounding environment for the presence ofpotential obstacles and can be similar to the radar 126 and/or laserrangefinder/lidar 128 in the vehicle 100.

The sensor unit 202 is mounted atop the vehicle 200 and includes one ormore sensors configured to detect information about an environmentsurrounding the vehicle 200, and output indications of the information.For example, sensor unit 202 can include any combination of cameras,radars, lidars, range finders, and acoustic sensors. The sensor unit 202can include one or more movable mounts that could be operable to adjustthe orientation of one or more sensors in the sensor unit 202. In oneembodiment, the movable mount could include a rotating platform thatcould scan sensors so as to obtain information from each directionaround the vehicle 200. In another embodiment, the movable mount of thesensor unit 202 could be movable in a scanning fashion within aparticular range of angles and/or azimuths. The sensor unit 202 could bemounted atop the roof of a car, although other mounting locations arepossible.

Additionally, the sensors of sensor unit 202 could be distributed indifferent locations and need not be collocated in a single location.Some possible sensor types and mounting locations include the twoadditional locations 216, 218. Furthermore, each sensor of sensor unit202 can be configured to be moved or scanned independently of othersensors of sensor unit 202.

In an example configuration, one or more radar scanners (e.g., first andsecond radar units 208, 210) can be located near the rear of the vehicle200, to actively scan the environment near the back of the vehicle 200for the presence of radio-reflective objects. Similarly, the firstlidar/radar unit 212 and the second lidar/radar unit 214 may be mountednear the front of the vehicle 200 to actively scan the environment nearthe front of the vehicle 200. A radar scanner can be situated, forexample, in a location suitable to illuminate a region including aforward-moving path of the vehicle 200 without occlusion by otherfeatures of the vehicle 200. For example, a radar scanner can beembedded in and/or mounted in or near the front bumper, frontheadlights, cowl, and/or hood, etc. Furthermore, one or more additionalradar scanning devices can be located to actively scan the side and/orrear of the vehicle 200 for the presence of radio-reflective objects,such as by including such devices in or near the rear bumper, sidepanels, rocker panels, and/or undercarriage, etc.

Although not shown in FIGS. 2A-2E, the vehicle 200 can include awireless communication system. The wireless communication system mayinclude wireless transmitters and receivers that could be configured tocommunicate with devices external or internal to the vehicle 200.Specifically, the wireless communication system could includetransceivers configured to communicate with other vehicles and/orcomputing devices, for instance, in a vehicular communication system ora roadway station. Examples of such vehicular communication systemsinclude DSRC, radio frequency identification (RFID), and other proposedcommunication standards directed towards intelligent transport systems.

The vehicle 200 can include a camera, possibly at a location insidesensor unit 202. The camera can be a photosensitive instrument, such asa still camera, a video camera, etc., that is configured to capture aplurality of images of the environment of the vehicle 200. To this end,the camera can be configured to detect visible light, and canadditionally or alternatively be configured to detect light from otherportions of the spectrum, such as infrared or ultraviolet light. Thecamera can be a two-dimensional detector, and can optionally have athree-dimensional spatial range of sensitivity. In some embodiments, thecamera can include, for example, a range detector configured to generatea two-dimensional image indicating distance from the camera to a numberof points in the environment. To this end, the camera may use one ormore range detecting techniques. For example, the camera can providerange information by using a structured light technique in which thevehicle 200 illuminates an object in the environment with apredetermined light pattern, such as a grid or checkerboard pattern anduses the camera to detect a reflection of the predetermined lightpattern from environmental surroundings. Based on distortions in thereflected light pattern, the vehicle 200 can determine the distance tothe points on the object. The predetermined light pattern may compriseinfrared light, or radiation at other suitable wavelengths for suchmeasurements. In some examples, the camera can be mounted inside a frontwindshield of the vehicle 200. Specifically, the camera can be situatedto capture images from a forward-looking view with respect to theorientation of the vehicle 200. Other mounting locations and viewingangles of camera can also be used, either inside or outside the vehicle200. Further, the camera can have associated optics operable to providean adjustable field of view. Still further, the camera can be mounted tovehicle 200 with a movable mount to vary a pointing angle of the camera,such as via a pan/tilt mechanism.

The vehicle 200 may include one or more other components in addition toor instead of those shown. The additional components may includeelectrical or mechanical functionality.

A control system of the vehicle 200 may be configured to control thevehicle 200 in accordance with a control strategy from among multiplepossible control strategies. The control system may be configured toreceive information from sensors coupled to the vehicle 200 (on or offthe vehicle 200), modify the control strategy (and an associated drivingbehavior) based on the information, and control the vehicle 200 inaccordance with the modified control strategy. The control systemfurther may be configured to monitor the information received from thesensors, and continuously evaluate driving conditions; and also may beconfigured to modify the control strategy and driving behavior based onchanges in the driving conditions.

FIG. 3 is a conceptual illustration of wireless communication betweenvarious computing systems related to an autonomous vehicle, according toexample embodiments. In particular, wireless communication may occurbetween remote computing system 302 and vehicle 200 via network 304.Wireless communication may also occur between server computing system306 and remote computing system 302, and between server computing system306 and vehicle 200.

Vehicle 200 can correspond to various types of vehicles capable oftransporting passengers or objects between locations, and may take theform of any one or more of the vehicles discussed above. In someinstances, vehicle 200 may operate in an autonomous mode that enables acontrol system to safely navigate vehicle 200 between destinations usingsensor measurements. When operating in an autonomous mode, vehicle 200may navigate with or without passengers. As a result, vehicle 200 maypick up and drop off passengers between desired destinations.

Remote computing system 302 may represent any type of device related toremote assistance techniques, including but not limited to thosedescribed herein. Within examples, remote computing system 302 mayrepresent any type of device configured to (i) receive informationrelated to vehicle 200, (ii) provide an interface through which a humanoperator can in turn perceive the information and input a responserelated to the information, and (iii) transmit the response to vehicle200 or to other devices. Remote computing system 302 may take variousforms, such as a workstation, a desktop computer, a laptop, a tablet, amobile phone (e.g., a smart phone), and/or a server. In some examples,remote computing system 302 may include multiple computing devicesoperating together in a network configuration.

Remote computing system 302 may include one or more subsystems andcomponents similar or identical to the subsystems and components ofvehicle 200. At a minimum, remote computing system 302 may include aprocessor configured for performing various operations described herein.In some embodiments, remote computing system 302 may also include a userinterface that includes input/output devices, such as a touchscreen anda speaker. Other examples are possible as well.

Network 304 represents infrastructure that enables wirelesscommunication between remote computing system 302 and vehicle 200.Network 304 also enables wireless communication between server computingsystem 306 and remote computing system 302, and between server computingsystem 306 and vehicle 200.

The position of remote computing system 302 can vary within examples.For instance, remote computing system 302 may have a remote positionfrom vehicle 200 that has a wireless communication via network 304. Inanother example, remote computing system 302 may correspond to acomputing device within vehicle 200 that is separate from vehicle 200,but with which a human operator can interact while a passenger or driverof vehicle 200. In some examples, remote computing system 302 may be acomputing device with a touchscreen operable by the passenger of vehicle200.

In some embodiments, operations described herein that are performed byremote computing system 302 may be additionally or alternativelyperformed by vehicle 200 (i.e., by any system(s) or subsystem(s) ofvehicle 200). In other words, vehicle 200 may be configured to provide aremote assistance mechanism with which a driver or passenger of thevehicle can interact.

Server computing system 306 may be configured to wirelessly communicatewith remote computing system 302 and vehicle 200 via network 304 (orperhaps directly with remote computing system 302 and/or vehicle 200).Server computing system 306 may represent any computing deviceconfigured to receive, store, determine, and/or send informationrelating to vehicle 200 and the remote assistance thereof. As such,server computing system 306 may be configured to perform anyoperation(s), or portions of such operation(s), that is/are describedherein as performed by remote computing system 302 and/or vehicle 200.Some embodiments of wireless communication related to remote assistancemay utilize server computing system 306, while others may not.

Server computing system 306 may include one or more subsystems andcomponents similar or identical to the subsystems and components ofremote computing system 302 and/or vehicle 200, such as a processorconfigured for performing various operations described herein, and awireless communication interface for receiving information from, andproviding information to, remote computing system 302 and vehicle 200.

The various systems described above may perform various operations.These operations and related features will now be described.

In line with the discussion above, a computing system (e.g., remotecomputing system 302, server computing system 306, or a computing systemlocal to vehicle 200) may operate to use a camera to capture images ofthe environment of an autonomous vehicle. In general, at least onecomputing system will be able to analyze the images and possibly controlthe autonomous vehicle.

In some embodiments, to facilitate autonomous operation a vehicle (e.g.,vehicle 200) may receive data representing objects in an environment inwhich the vehicle operates (also referred to herein as “environmentdata”) in a variety of ways. A sensor system on the vehicle may providethe environment data representing objects of the environment. Forexample, the vehicle may have various sensors, including a camera, aradar unit, a laser range finder, a microphone, a radio unit, and othersensors. Each of these sensors may communicate environment data to aprocessor in the vehicle about information each respective sensorreceives.

In one example, a camera may be configured to capture still imagesand/or video. In some embodiments, the vehicle may have more than onecamera positioned in different orientations. Also, in some embodiments,the camera may be able to move to capture images and/or video indifferent directions. The camera may be configured to store capturedimages and video to a memory for later processing by a processing systemof the vehicle. The captured images and/or video may be the environmentdata. Further, the camera may include an image sensor as describedherein.

In another example, a radar unit may be configured to transmit anelectromagnetic signal that will be reflected by various objects nearthe vehicle, and then capture electromagnetic signals that reflect offthe objects. The captured reflected electromagnetic signals may enablethe radar system (or processing system) to make various determinationsabout objects that reflected the electromagnetic signal. For example,the distances to and positions of various reflecting objects may bedetermined. In some embodiments, the vehicle may have more than oneradar in different orientations. The radar system may be configured tostore captured information to a memory for later processing by aprocessing system of the vehicle. The information captured by the radarsystem may be environment data.

In another example, a laser range finder may be configured to transmitan electromagnetic signal (e.g., infrared light, such as that from a gasor diode laser, or other possible light source) that will be reflectedby target objects near the vehicle. The laser range finder may be ableto capture the reflected electromagnetic (e.g., laser) signals. Thecaptured reflected electromagnetic signals may enable the range-findingsystem (or processing system) to determine a range to various objects.The laser range finder may also be able to determine a velocity or speedof target objects and store it as environment data.

Additionally, in an example, a microphone may be configured to captureaudio of environment surrounding the vehicle. Sounds captured by themicrophone may include emergency vehicle sirens and the sounds of othervehicles. For example, the microphone may capture the sound of the sirenof an ambulance, fire engine, or police vehicle. A processing system maybe able to identify that the captured audio signal is indicative of anemergency vehicle. In another example, the microphone may capture thesound of an exhaust of another vehicle, such as that from a motorcycle.A processing system may be able to identify that the captured audiosignal is indicative of a motorcycle. The data captured by themicrophone may form a portion of the environment data.

In yet another example, the radio unit may be configured to transmit anelectromagnetic signal that may take the form of a Bluetooth signal,802.11 signal, and/or other radio technology signal. The firstelectromagnetic radiation signal may be transmitted via one or moreantennas located in a radio unit. Further, the first electromagneticradiation signal may be transmitted with one of many differentradio-signaling modes. However, in some embodiments it is desirable totransmit the first electromagnetic radiation signal with a signalingmode that requests a response from devices located near the autonomousvehicle. The processing system may be able to detect nearby devicesbased on the responses communicated back to the radio unit and use thiscommunicated information as a portion of the environment data.

In some embodiments, the processing system may be able to combineinformation from the various sensors in order to make furtherdeterminations of the environment of the vehicle. For example, theprocessing system may combine data from both radar information and acaptured image to determine if another vehicle or pedestrian is in frontof the autonomous vehicle. In other embodiments, other combinations ofsensor data may be used by the processing system to make determinationsabout the environment.

While operating in an autonomous mode, the vehicle may control itsoperation with little-to-no human input. For example, a human-operatormay enter an address into the vehicle and the vehicle may then be ableto drive, without further input from the human (e.g., the human does nothave to steer or touch the brake/gas pedals), to the specifieddestination. Further, while the vehicle is operating autonomously, thesensor system may be receiving environment data. The processing systemof the vehicle may alter the control of the vehicle based on environmentdata received from the various sensors. In some examples, the vehiclemay alter a velocity of the vehicle in response to environment data fromthe various sensors. The vehicle may change velocity in order to avoidobstacles, obey traffic laws, etc. When a processing system in thevehicle identifies objects near the vehicle, the vehicle may be able tochange velocity, or alter the movement in another way.

When the vehicle detects an object but is not highly confident in thedetection of the object, the vehicle can request a human operator (or amore powerful computer) to perform one or more remote assistance tasks,such as (i) confirm whether the object is in fact present in theenvironment (e.g., if there is actually a stop sign or if there isactually no stop sign present), (ii) confirm whether the vehicle'sidentification of the object is correct, (iii) correct theidentification if the identification was incorrect and/or (iv) provide asupplemental instruction (or modify a present instruction) for theautonomous vehicle. Remote assistance tasks may also include the humanoperator providing an instruction to control operation of the vehicle(e.g., instruct the vehicle to stop at a stop sign if the human operatordetermines that the object is a stop sign), although in some scenarios,the vehicle itself may control its own operation based on the humanoperator's feedback related to the identification of the object.

To facilitate this, the vehicle may analyze the environment datarepresenting objects of the environment to determine at least one objecthaving a detection confidence below a threshold. A processor in thevehicle may be configured to detect various objects of the environmentbased on environment data from various sensors. For example, in oneembodiment, the processor may be configured to detect objects that maybe important for the vehicle to recognize. Such objects may includepedestrians, street signs, other vehicles, indicator signals on othervehicles, and other various objects detected in the captured environmentdata.

The detection confidence may be indicative of a likelihood that thedetermined object is correctly identified in the environment, or ispresent in the environment. For example, the processor may performobject detection of objects within image data in the receivedenvironment data, and determine that the at least one object has thedetection confidence below the threshold based on being unable toidentify the object with a detection confidence above the threshold. Ifa result of an object detection or object recognition of the object isinconclusive, then the detection confidence may be low or below the setthreshold.

The vehicle may detect objects of the environment in various waysdepending on the source of the environment data. In some embodiments,the environment data may come from a camera and be image or video data.In other embodiments, the environment data may come from a lidar unit.The vehicle may analyze the captured image or video data to identifyobjects in the image or video data. The methods and apparatuses may beconfigured to monitor image and/or video data for the presence ofobjects of the environment. In other embodiments, the environment datamay be radar, audio, or other data. The vehicle may be configured toidentify objects of the environment based on the radar, audio, or otherdata.

In some embodiments, the techniques the vehicle uses to detect objectsmay be based on a set of known data. For example, data related toenvironmental objects may be stored to a memory located in the vehicle.The vehicle may compare received data to the stored data to determineobjects. In other embodiments, the vehicle may be configured todetermine objects based on the context of the data. For example, streetsigns related to construction may generally have an orange color.Accordingly, the vehicle may be configured to detect objects that areorange, and located near the side of roadways as construction-relatedstreet signs. Additionally, when the processing system of the vehicledetects objects in the captured data, it also may calculate a confidencefor each object.

Further, the vehicle may also have a confidence threshold. Theconfidence threshold may vary depending on the type of object beingdetected. For example, the confidence threshold may be lower for anobject that may require a quick responsive action from the vehicle, suchas brake lights on another vehicle. However, in other embodiments, theconfidence threshold may be the same for all detected objects. When theconfidence associated with a detected object is greater than theconfidence threshold, the vehicle may assume the object was correctlyrecognized and responsively adjust the control of the vehicle based onthat assumption.

When the confidence associated with a detected object is less than theconfidence threshold, the actions that the vehicle takes may vary. Insome embodiments, the vehicle may react as if the detected object ispresent despite the low confidence level. In other embodiments, thevehicle may react as if the detected object is not present.

When the vehicle detects an object of the environment, it may alsocalculate a confidence associated with the specific detected object. Theconfidence may be calculated in various ways depending on theembodiment. In one example, when detecting objects of the environment,the vehicle may compare environment data to predetermined data relatingto known objects. The closer the match between the environment data andthe predetermined data, the higher the confidence. In other embodiments,the vehicle may use mathematical analysis of the environment data todetermine the confidence associated with the objects.

In response to determining that an object has a detection confidencethat is below the threshold, the vehicle may transmit, to the remotecomputing system, a request for remote assistance with theidentification of the object. As discussed above, the remote computingsystem may take various forms. For example, the remote computing systemmay be a computing device within the vehicle that is separate from thevehicle, but with which a human operator can interact while a passengeror driver of the vehicle, such as a touchscreen interface for displayingremote assistance information. Additionally or alternatively, as anotherexample, the remote computing system may be a remote computer terminalor other device that is located at a location that is not near thevehicle.

The request for remote assistance may include the environment data thatincludes the object, such as image data, audio data, etc. The vehiclemay transmit the environment data to the remote computing system over anetwork (e.g., network 304), and in some embodiments, via a server(e.g., server computing system 306). The human operator of the remotecomputing system may in turn use the environment data as a basis forresponding to the request.

In some embodiments, when the object is detected as having a confidencebelow the confidence threshold, the object may be given a preliminaryidentification, and the vehicle may be configured to adjust theoperation of the vehicle in response to the preliminary identification.Such an adjustment of operation may take the form of stopping thevehicle, switching the vehicle to a human-controlled mode, changing avelocity of vehicle (e.g., a speed and/or direction), among otherpossible adjustments.

In other embodiments, even if the vehicle detects an object having aconfidence that meets or exceeds the threshold, the vehicle may operatein accordance with the detected object (e.g., come to a stop if theobject is identified with high confidence as a stop sign), but may beconfigured to request remote assistance at the same time as (or at alater time from) when the vehicle operates in accordance with thedetected object.

FIG. 4A is an illustration of a camera 400, according to exampleembodiments. The camera 400 may include one or more lenses housed withina lens barrel of the camera 400. The camera 400 may also include animage sensor 404. The image sensor 404 may receive light from the scenevia the one or more lenses housed within the lens barrel and a mirror401. In some embodiments, the camera 400 may also include additionalcomponents (e.g., shutter buttons, viewfinders, flashes, batteries,electronic storage for recording captured images, display screens,selection buttons, etc.). Further, in some embodiments, the mirror 401may move (e.g., the mirror 401 may rotate relative to the lens such thatlight from the scene can be selectively directed to a viewfinder of thecamera 400 or to the image sensor 404 of the camera 400).

In some embodiments, one or more of the camera 400 components may becontrolled manually by a user of the camera 400. For example, the lensbarrel may be configured to rotate about its axis to modify the relativepositions of the one or more lenses within the lens barrel, therebyadjusting a field of view and/or a zoom of the camera 400.Alternatively, one or more of the components of the camera 400 may beelectronically controlled (e.g., a camera controller may adjust one ormore of the lenses within the lens barrel to modify a zoom of thecamera, such as during an auto-focus procedure).

While a digital single-lens reflex (DSLR) camera is illustrated in FIG.4A, it is understood that FIG. 4A is solely provided as an example. Inalternate embodiments, other form factors may be used. For example, insome embodiments, the camera 400 may only include an image sensor behindone or more lenses (e.g., a telecentric lens). Other arrangements arealso possible. In some embodiments, for instance, the camera may includeone or more optical filters (e.g., polarization filters, chromaticfilters, neutral-density filters, etc.) and/or one or more electronicstages and/or motors configured to adjust the position of one or morecomponents of the camera. Further, in some embodiments, the camera 400may be used for object detection and avoidance within an autonomousvehicle (e.g., like the camera 130 illustrated and described withreference to FIG. 1). In some embodiments, the camera 400 may beadversely impacted by stray light incident on the image sensor 404(e.g., based on the mechanisms described with reference to FIGS. 4B-4D).It is understood that the techniques described herein for detectingand/or mitigating the stray light may be applicable to any form factorof camera and/or image sensor. For example, the techniques describedherein may determine an amount and/or location of stray light incidenton an image sensor within an autonomous vehicle camera (e.g., with oneor more associated camera optics), a webcam, a cellphone camera, a DSLR(as illustrated in FIG. 4A), a backup camera on a vehicle, aclosed-circuit television camera, a highspeed camera, etc. Additionally,the techniques described herein may equally be applied to cameras thatdigitally record images as well as cameras that chemically record images(e.g., on film).

FIG. 4B illustrates an optical detector system 410. The optical detectorsystem 410 may be a component of a camera (e.g., the camera 130 shownand described with reference to FIG. 1 and/or the camera 400 illustratedin FIG. 4A). In some embodiments, the optical detector system 410 may beused to capture images of an environment surrounding a vehicle (e.g.,the vehicle 200 illustrated in FIG. 2). Such images may be captured forobject detection and avoidance, to determine the amount of stray lightincident on the image sensor 404, or for both object detection andavoidance and to determine the amount of stray light incident on theimage sensor 404. As illustrated, the optical detector system 410 mayinclude a body 402 (e.g., which may include a lens barrel), an imagesensor 404, a first lens 412, and a second lens 414.

The body 402 may provide a rigid housing for the rest of the componentsof the optical detector system 410. Further, the rest of the componentsin the optical detector system 410 may be attached to the housing so asto provide a proper alignment and relative position of the remainingcomponents. For example, the first lens 412 and the second lens 414 maybe attached to a lens-barrel portion of the body 402 such that the firstlens 412 and the second lens 414 are separated by a predetermineddistance and such that the first lens 412 and the second lens 414 areproperly aligned with one another and the image sensor 404.Additionally, in some embodiments, the inner surface of the body 402 maybe made of and/or coated with an anti-reflective material (e.g.,blackened steel) to mitigate internal reflections. In addition to orinstead of using an anti-reflective material on an inner surface of thebody 402, in some embodiments, the first lens 412 and/or the second lens414 may be coated with an anti-reflective material to mitigate internalreflections.

FIGS. 4B-4D are provided as examples. It is understood that in alternateembodiments contemplated herein, the optical detector system 410 mayinclude greater or fewer lenses (e.g., three lenses, four lenses, fivelenses, etc.), lenses with different orientations, lenses of differentstyles (e.g., as opposed to the plano-convex first lens 412 and theconvex second lens 414 illustrated), different separations of lenses, orlenses with different focal lengths. Further, it is understood that, insome embodiments, the optical detector system 410 may include additionalor alternative imaging optics. For example, the optical detector system410 may include one or more mirrors, one or more apertures, one or moreoptical filters, etc. In some embodiments, the optical detector system410 may include an adjustable focus arrangement (e.g., a set of lensesthat may be translated with respect to one another adjust an associatedfocal point). Such adjustable focus arrangements may be manuallyadjusted or adjusted in an automated fashion (e.g., by a controllerconfigured to reorient/translate stages on which each of the lenses ismounted; the controller may adjust the focus of the lens arrangementbased on the quality of a captured image, e.g., to autofocus the opticaldetector system 410).

The image sensor 404 may be configured to capture an image of theenvironment surrounding the optical detector system 410 (i.e., thescene) by intercepting light signals from the environment via the firstlens 412 and the second lens 414. The image sensor 404 may includevarious light detectors or detector arrays to enable the detection ofsuch light signals. For example, in various embodiments, the imagesensor 404 may include an array of single photon avalanche detectors(SPADs), an array of avalanche photodiodes (APDs), one or more siliconphotomultipliers (SiPMs), an array of photodiodes, an array ofphototransistors, an array of active pixel sensors (APSs), one or moreCCDs, one or more cryogenic detectors, etc. In some embodiments, theoptical detector system 410 may also include or be connected to a memory(e.g., a non-transitory, computer-readable medium, such as a hard drive)configured to store (e.g., as a series of digital bits in a pixel bypixel arrangement) images captured by the image sensor 404.

Optical detector systems, such as the optical detector system 410illustrated in FIG. 4B, can be susceptible to internal reflections(e.g., which may appear on the image sensor 404 as stray light and/or aghost image). For example, imperfections (e.g., impurities, cracks, airbubbles, scratches, dirt, dust, bug splatters, degradation of lenscoatings, condensation, debris, imperfect transparency, etc.) within oron one of the lenses can lead to additional light signals (i.e., straylight signals) intersecting the image sensor 404 based on a singleprimary light signal. As illustrated in FIG. 4B, a primary light signal422 may be transmitted by an object 420 (e.g., a bright object, such asthe sun, that has high intensity relative to the other objects in thescene) in the surrounding environment of the optical detector system410. Upon entering the optical detector system 410 and interacting withthe first lens 412, for example, the primary light signal 422 may splitinto a reduced-intensity primary light signal 424, secondary lightsignals 426, and tertiary light signals 428. The reduced-intensityprimary light signal 424, the secondary light signals 426, and thetertiary light signals 428 may be different intensities from oneanother.

In other embodiments, the primary light signal 422 may separate intogreater or fewer additional light signals upon interacting with thefirst lens 412. Additionally or alternatively, any light signalsgenerated based on the interaction of the primary light signal 422 withthe first lens 412 may further generate additional light signals basedon interactions with additional lenses in the optical detector system410 (e.g., in some embodiments, the secondary light signals 426 or thetertiary light signals 428 may each further split into multiple lightsignals based on their interactions with the second lens 414). It isunderstood that the angles of the secondary light signals 426 and thetertiary light signals 428 relative to the reduced-intensity primarylight signal 424 and the primary light signal 422 are illustrated onlyas examples. In other embodiments, the relative angles of the secondarylight signals 426 or the tertiary light signals 428 may be different(e.g., based on the wavelength(s) of the primary light signal 422, thematerial composition of the first lens 412, the respective defects inthe first lens 412, the ambient temperature, the ambient air pressure,the lens shape of the first lens 412, etc.).

As a result of the additional light signals (e.g., the secondary lightsignals 426 and the tertiary light signals 428) generated by theinteraction of the primary light signal 422 with the first lens 412, animage captured by the image sensor 404 of the surrounding environmentmay be distorted and/or imprecise. In some embodiments, the object 420may appear to be a different shape, size, and/or intensity than itotherwise would as a result of such imperfections in the opticaldetector system 410. For example, a veiling glare, lens flare, or ghostimages (each being forms of stray light) may be present in an image ofthe object 420 captured by the image sensor 404, thereby obscuring theobject 420 within the captured image or obscuring other portions of thecaptured image. This may prevent observation of relatively faint objectsthat are positioned close to bright objects within the captured image(e.g., a pedestrian standing in front of the sun relative to the opticaldetector system 410 may be washed out by the sun, particularly if theveiling glare generated in the optical detector system 410 by the sun issevere). Further, such veiling glare, lens flare, or ghost images canadversely affect an ability of the image sensor 404 to capture imageswith a high dynamic range.

Other optical mechanisms can give rise to images captured by the imagesensor 404 that do not precisely reflect the surrounding scene (e.g.,can give rise to veiling glare, lens flare, or ghost images). Forexample, primary light signals 422 from objects 420 may be reflectedfrom the incident face of one or more of the lens. For instance, asillustrated in FIG. 4C, the primary light signal 422 may be reflected bythe front face of the second lens 414 to generate a reflected lightsignal 432. The reflected light signal 432 may then reflect off of theback face of the first lens 412, be transmitted through the second lens414, and then intersect with the image sensor 404, as illustrated. Aswith the mechanism illustrated in FIG. 4B, the mechanism illustrated inFIG. 4C could result in veiling glare, lens flare, or ghost images thatobscure objects within images captured by the image sensor 404 or causethe source object 420 to have an inaccurate intensity, shape, or size.It is understood that the angle of the reflected light signal 432relative to the reduced-intensity primary light signal 424 and theprimary light signal 422 are illustrated only as an example. In otherembodiments, the relative angle of the reflected light signal 432 may bedifferent (e.g., based on the wavelength(s) of the primary light signal422, the material composition of the second lens 414, the respectivedefects in the second lens 414, the ambient temperature, the ambient airpressure, the lens shape of the second lens 414, the position of theobject 420 relative to the optical detector system 410, etc.). Further,in some embodiments, there may be multiple reflected light signalsgenerated based on the primary light signal 422.

Another optical mechanism that can give rise to stray light in capturedimages is illustrated in FIG. 4D. As illustrated, the primary lightsignal 422 may be reflected off of an interior surface of the body 402(e.g., a lens-barrel portion of the body 402) to generate a reflectedlight signal 442. Also as illustrated, in some embodiments, the primarylight signal 422 may be partially reflected by the interior surface ofthe body 402 and partially absorbed by the interior surface of the body402, resulting in a reflected light signal 442 that has a reducedintensity compared with the primary signal 422. After being generated,the reflected light signal 442 may then be transmitted to the imagesensor 404. As with the mechanisms illustrated in FIGS. 4B and 4C, themechanism illustrated in FIG. 4D could obscure objects within imagescaptured by the image sensor 404. Further, the mechanism illustrated inFIG. 4D could lead to light from objects that would otherwise not becaptured to be captured by the image sensor 404 (i.e., objects outsideof a field of view of the optical detector system 410 may produce lightthat is nonetheless captured by the image sensor 404). It is understoodthat the angle of the reflected light signal 442 relative to the primarylight signal 422 is illustrated only as an example. In otherembodiments, the relative angle of the reflected light signal 442 may bedifferent (e.g., based on the wavelength(s) of the primary light signal422, the material composition of the inner surface of the body 402, thematerial composition of a coating on the inner surface of the body 402,the ambient temperature, the ambient air pressure, the position of theobject 420 relative to the optical detector system 410, the shape of alens-barrel portion of the body 402, etc.). Further, in someembodiments, there may be multiple reflected light signals generatedbased on the primary light signal 422.

As illustrated in FIGS. 4B-4D, some stray light may intersect the imagesensor 404 at a large angle of incidence relative to the surface of theimage sensor 404 and some stray light may intersect the image sensor 404with a small angle of incidence relative to a normal vector extendingfrom the surface of the image sensor 404. In alternate embodiments, itis possible that stray light only insects the image sensor 404 at largeangles of incidence or only at small angles of incidence. Further, whilestray light might intersect the image sensor 404 at either small orlarge angles (as illustrated in FIGS. 4B-4D), object light from thesurrounding environment that is to be imaged may intersect the imagesensor 404 at small angles only. Hence, any light that intersects theimage sensor 404 at a large angle of incidence relative to a normalvector extending from the surface of the image sensor 404 may correspondto stray light, whereas there may be ambiguity regarding light thatintersects the image sensor 404 at a small angle of incidence as towhether that light corresponds to object light or stray light. Suchinsight can be used to identify and/or eliminate stray light withinimages captured by the image sensor 404 using the techniques describedherein.

Based on the mechanisms described above and illustrated in FIGS. 4B-4D,stray light can be generated within the optical detector system 410based on light signals generated by objects 420 within the environmentsurrounding the optical detector system 410. FIG. 5 illustrates an imagecaptured by an image sensor 404 of an optical detector system 410 thathas veiling glare and lens flare present (e.g., as a result of one ormore of the mechanisms described with reference to FIGS. 4B-4D). Such animage may be referred to herein as a captured image 502 with glare.Other types of stray light (besides veiling glare and lens flare) withincaptured images (e.g., ghost images of one or more objects in theenvironment) are also possible.

As a result of the generated glare, regions of the captured image 502may have locations of higher intensity than they otherwise would havewithout such glare. For example, the captured image 502 hashigh-intensity regions near the position of the sun within the capturedimage 502, as illustrated. Such artificially increased intensity regionsmay adversely affect object detection and/or identification (e.g., forobject detection and avoidance within an autonomous vehicle).

In some embodiments, the optical detector system 410 used to captureimages upon which stray light evaluations are made may be an opticaldetector system 410 used for object detection and avoidance. Forexample, the captured images may be images that can be used to identifycertain objects within a scene, but can also be used to identify one ormore locations of stray light incident upon the image sensor 404.Additionally or alternatively, one or more images may be captured solelyto identify locations of stray light incident on the image sensor 404.Thereafter, stray light may be mitigated for later captured images(e.g., in post-processing of those captured images based on the straylight determination image and/or by employing one or more stray lightmitigation optics, such as a neutral-density filter). In extreme casesof stray light, the camera associated with the optical detector system410 may be decommissioned (e.g., if it is determined that the straylight incident on the image sensor 404 cannot be accounted for in futureimages and if the stray light is so egregious that the captured imagescould not be used for object detection and avoidance within acorresponding autonomous vehicle).

FIG. 6A is an illustration of an image sensor (e.g., the image sensor404 illustrated in FIGS. 4A-4D). The image sensor 404 is shown in atop-view perspective (e.g., from a perspective along the positivez-axis, as illustrated). FIG. 6B shows a cutaway of the image sensor 404from a side-view perspective (e.g., a cross-section taken alongthey-axis). The image sensor 404 may include a plurality oflight-sensitive pixels 604 mounted into an image sensor mount 602. Thelight-sensitive pixels 604 may detect light signals from a surroundingenvironment to produce an image (e.g., each of the light-sensitivepixels 604 may convert optical energy into electrical energy that isstored within a memory of a computing device and the intensities of theelectrical energies from each pixel may be arranged to form an image).

Further, the image sensor 404 may include a plurality of microlenses606. Each of the microlenses 606 may be positioned over a respectivelight-sensitive pixel 604 to maximize the amount of signal from thesurrounding environment that is directed to the light-sensitive portionof each light-sensitive pixel 604. For example, the microlenses 606 mayconically focus any light incident on the face of the microlens 606 ontoa detection surface of the light-sensitive pixel 604. The microlenses606 may be shaped as bubbles or other convex structures, in someembodiments.

As illustrated in FIG. 6B, in some embodiments the image sensor mount602 may cover one or more of the light-sensitive pixels 604 (e.g., mayocclude one or more of the light-sensitive pixels 604 located around aperiphery of the image sensor 404). As such, signals from thelight-sensitive pixels 604 covered by the image sensor mount 602 may beremoved from captured images in post-processing. The image sensor 404illustrated in FIGS. 6A and 6B may not be capable of determining thepresence of stray light, unlike the image sensors described below withreference to FIGS. 7A-10, for example.

FIG. 7A is an illustration of an image sensor 704, according to exampleembodiments. The image sensor 704 is shown in a top-view perspective(e.g., from a perspective along the positive z-axis, as illustrated).FIG. 7B shows a cutaway of the image sensor 704 from a side-viewperspective (e.g., a cross-section taken along the y-axis). Similar tothe image sensor 404 illustrated in FIGS. 6A and 6B, the image sensor704 may include a plurality of light-sensitive pixels 604 mounted intoan image sensor mount 602 and an associated plurality of microlenses606. However, unlike the image sensor 404 of FIGS. 6A and 6B, the imagesensor 704 of FIGS. 7A and 7B may include a plurality of stray-lightmicrolenses 712. The stray-light microlenses 712 may be used to detectthe presence of stray light incident on the light-sensitive surfaces ofthe light-sensitive pixels 604. To accomplish this, in some embodiments,the stray-light microlenses 712 may be conically shaped (e.g., with aconvex outer surface), hemispherically shaped, bubble-shaped, orotherwise convexly shaped so as to focus light signals that intersect asurface of the stray-light microlens 712. It is understood that FIGS. 7Aand 7B are examples and not necessarily to-scale. Further, in someembodiments, a greater percentage of the surface area of the imagesensor 404 may be occupied by microlenses 606 compared to stray-lightmicrolenses 712 than is illustrated in FIGS. 7A and 7B (e.g., there maybe fewer stray-light microlenses 712 compared to other microlenses 606than is illustrated).

The microlenses 606 and/or the stray-light microlenses 712 may befabricated using a variety of techniques. For example, the microlenses606 and/or the stray-light microlenses 712 may be fabricated usingphotolithography into curable epoxy (e.g., ultraviolet curable epoxy)and/or photoresist and then melting the polymer to form an array.Alternatively, the microlenses 606 and/or the stray-light microlenses712 may be fabricated using molding or embossing from a mold orreplication of an electroform using a master mandrel of the array.

In some embodiments, as illustrated in FIGS. 7A and 7B, the stray-lightmicrolenses 712 may be larger than the rest of the microlenses 606(e.g., may be positioned over two, four, nine, sixteen, etc.light-sensitive pixels 604 in a 1×1 array, a 2×2 array, a 3×3 array, a4×4 array, etc., respectively, or in a non-square array oflight-sensitive pixels). Also as illustrated, in place of themicrolenses 606, the stray-light microlenses 712 may be positioned overthe corresponding subset of light-sensitive pixels 604. Further, asillustrated in FIG. 7A, the stray-light microlenses 712 and thecorresponding underlying subsets of light-sensitive pixels 604 may bepositioned along a periphery of the image sensor 704 (e.g., in a cornerof the image sensor 704, as illustrated). It is understood that this issolely provided as an example. In alternate embodiments (e.g., asillustrated in FIGS. 9A-10B), the stray-light microlenses 712 andcorresponding underlying subsets of light-sensitive pixels 604 may bepositioned in other regions of the image sensor 704. Additionally, insome embodiments, one or more of the stray-light microlenses 712 may bepositioned partially or wholly underneath a mount of the image sensor704 (e.g., an overhanging image sensor mount similar to the image sensormount 602 illustrated in FIGS. 6A and 6B). In this way, light-sensitivepixels 604 that would otherwise go unused for detecting a surroundingscene could be used to detect stray light incident on the image sensor704 using an associated stray-light microlens 712.

In alternate embodiments, one or more of the stray-light microlenses 712may be the same size as the rest of the microlenses 606 and/or smallerthan the rest of the microlenses 606. Further, in some embodiments,different stray-light microlenses corresponding to a single image sensormay be different sizes and/or sizes (e.g., one stray-light microlens maybe larger than another stray-light microlens and/or one stray-lightmicrolens may have a different associated focal plane than anotherstray-light microlens).

As illustrated in FIGS. 7A and 7B, the image sensor 704 may include aplurality of stray-light microlenses 712 over a corresponding pluralityof subsets of light-sensitive pixels 604. While each of the stray-lightmicrolenses 712 in FIG. 7A is the same size and shape and overlays thesame number of corresponding light-sensitive pixels 604, it isunderstood that, in some embodiments, one or more of the stray-lightmicrolenses 712 could have a different size and/or shape from one ormore of the other stray-light microlenses 712.

The stray-light microlenses 712 may direct light from a receivingsurface (e.g., a convex surface) of the stray-light microlens 712 ontoone or more of the underlying light-sensitive pixels 604. This isillustrated in FIGS. 7C-7F, for example. FIG. 7C illustrates a singlestray-light microlens 712 overlaying a subset of nine light-sensitivepixels 604 from a top-view perspective (e.g., from a perspective alongthe positive z-axis, as illustrated). FIG. 7D shows a cutaway of thesame thing (e.g., a cross-section taken along the y-axis). Alsoillustrated in FIGS. 7C and 7D is a light signal 722 incident on thestray-light microlens 712 (e.g., the resulting spot generated by thelight signal 722 intersecting the surface of the light-sensitive pixels604 is illustrated in FIG. 7C and the incoming light signal 722 isillustrated in FIG. 7D). Because the light signal 722 is incidentperpendicularly to or approximately perpendicularly to a surface of thelight-sensitive pixels 604 (i.e., at a low angle, such as lower than15°, of incidence relative to a vector extending perpendicularly from asurface of the light-sensitive pixels 604), the stray-light microlens712 may focus the light signal 722 onto a middle region of theunderlying light-sensitive pixels 604. For example, as illustrated inFIGS. 7C and 7D, the light signal 722 is directed to a center portion ofthe detection surface of the middle of nine underlying light-sensitivepixels 604.

It is understood that FIGS. 7C and 7D are provided solely as examples.In some embodiments, a light signal incident perpendicularly to orapproximately perpendicularly to a surface of the light-sensitive pixels604 may be spread across all of the underlying light-sensitive pixels604, but in unequal intensities (e.g., a larger intensity onlight-sensitive pixels 604 near a center of the underlying subset oflight-sensitive pixels 604 and smaller intensities on peripherallight-sensitive pixels 604 of the underlying subset of light-sensitivepixels 604). Regardless of the exact distribution, though, the lightsignal will be distributed to a predetermined region of the underlyinglight-sensitive pixels 604 when incident perpendicularly to orapproximately perpendicularly to a surface of the light-sensitive pixels604.

Similarly, when a light signal is obliquely incident to a surface of thelight-sensitive pixels 604 (e.g., at a high angle, such as greater than15°, of incidence relative to a vector extending perpendicularly from asurface of the light-sensitive pixels 604), the stray-light microlens712 may distribute the light signal across a region of the underlyinglight-sensitive pixels 604 that is away from a center region of theunderlying light-sensitive pixels 604. This is illustrated in FIGS. 7Eand 7F, for example. Like FIG. 7C, FIG. 7E illustrates the stray-lightmicrolens 712 overlaying the subset of nine light-sensitive pixels 604from a top-view perspective (e.g., from a perspective along the positivez-axis, as illustrated). Similarly, FIG. 7F, like FIG. 7D, shows acutaway (e.g., a cross-section taken along the y-axis) of thestray-light microlens 712 and the light-sensitive pixels 604.

Unlike FIGS. 7C and 7D, though, FIGS. 7E and 7F illustrate an obliquelight signal 732 incident on the stray-light microlens 712 (e.g., theresulting spot generated by the oblique light signal 732 intersectingthe surface of the light-sensitive pixels 604 is illustrated in FIG. 7Eand the incoming oblique light signal 732 is illustrated in FIG. 7F). Asillustrated in FIGS. 7E and 7F, when the oblique light signal 732contacts a surface of the stray-light microlens 712, the oblique lightsignal 732 may be directed to a peripheral portion of the surface oflight-sensitive pixels 604 (e.g., a light-sensitive pixel 604 along anedge of the subset of light-sensitive pixels 604 underneath thestray-light microlens 712 may receive all or a majority of the intensityof the oblique light signal 732).

Based on the different intensity distributions associated with theincoming light signals 722, 732 illustrated in FIGS. 7C and 7D whencompared to FIGS. 7E and 7F, a determination of an angle of incidence ofthe light signal can be made by comparing the relative intensitiesdetected by each light-sensitive pixel 604 in the subset oflight-sensitive pixels 604 underlying the respective stray-lightmicrolens 712. For example, a processor (e.g., a controller associatedwith the image sensor 704, a controller associated with a camera ofwhich the image sensor 704 is a component, a controller associated withan autonomous vehicle of which the image sensor 704 is a component, or aserver computing device to which the image sensor 704 has communicated acaptured image) may determine a first angle of incidence of a firstlight signal detected by the subset of light-sensitive pixels 604underlying the stray-light microlens 712.

Because higher angles of incidence are more frequently associated withstray light than with a light signal from an object in the surroundingenvironment, once a determination of an angle of incidence is made, adetermination can be made about whether the incident light signalconstitutes stray light. For example, a processor (e.g., a controllerthat made a determination about the angle of incidence of one or morelight signals detected by the light-sensitive pixels 604) may determinean amount of stray light incident on the image sensor 704 (e.g.,incident on the light-sensitive pixels 604 of the image sensor 704underlying the stray-light microlens 712) based on the determined angleof incidence. This determination may be made by comparing a determinedangle of incidence to a threshold angle of incidence (e.g., about 5°,about 10°, about 15°, about 20°, about 25°, about 30°, etc.). Thethreshold angle of incidence may be measured relative to a vector thatextends perpendicularly from a surface of the image sensor 704. If thedetermined angle of incidence is greater than the threshold angle, theincident light signal may be identified as a stray light signal. Thethreshold angle may be based on the geometry and/or materials of theimage sensor 704 and/or the geometry and/or materials of the stray-lightmicrolenses 712. Alternatively, the threshold angle may be determined bythe processor based on one or more environmental factors and/or may beset by a user of the camera (e.g., a user may set a lower thresholdangle of incidence to identify more light as stray light). In someembodiments, the underlying light-sensitive pixels 604 maysimultaneously detect stray light signals and non-stray light signals.Hence, a determination could also be made regarding what percentage ofthe light intensity incident on the underlying subset of light-sensitivepixels 604 corresponds to stray light and what percentage corresponds toobject light. For example, if 50% of the received light were incident onthe center light-sensitive pixel 604 underlying the stray-lightmicrolens 712 and 50% of the received light were incident on aperipheral light-sensitive pixel 604 underlying the stray-lightmicrolens 712, it may be determined that 50% of the light received atthat portion of the image sensor 704 corresponds to stray light.

Each of the stray-light microlenses 712 illustrated in FIG. 7A may beused to make a similar determination about stray light incident on thelight-sensitive pixels 604 underlying the respective stray-lightmicrolens 712 (e.g., a processor may determine an amount of stray lightincident on the light-sensitive pixels 604 underlying each of thestray-light microlenses 712 based on the determined angle of incidenceof the light on the underlying light-sensitive pixels 604). In someembodiments, for example, a controller may determine an angle ofincidence for each light signal detected by each subset oflight-sensitive pixels 604 underlying each stray-light microlens 712.Then, based on the determined angles of incidence, the controller maydetermine an amount of stray light incident on the image sensor 704.

The amount of stray light incident on the image sensor 704 may includean amount incident on the respective regions of the image sensor 704where the stray-light microlenses 712 are located, an aggregate quantityrepresenting an amount of stray light incident across the entire imagesensor 704 (e.g., based on extrapolation of the amount of stray lightincident on the respective regions of the image sensor 704 where thestray-light microlenses 712 are located), and/or a map (e.g., a plot) ofstray light across the image sensor 704 (e.g., based on interpolation ofthe amount of stray light incident on the respective regions of theimage sensor 704 where the stray-light microlenses 712 are located).

In some embodiments, the controller may also determine a true-blackoptical level (and/or a true-black electrical level) based on thedetermined amount of stray light incident on the image sensor 704. Forexample, the controller may perform a regression analysis (e.g., ofdetermined stray light vs. detected object signal) or statisticalanalysis of the stray light incident on each subset of light-sensitivepixels 604 that is beneath a stray-light microlens 712. Then, based onthis analysis, the controller may determine ay-intercept (e.g., anintercept along a stray-light intensity axis when detected object signalis equal to 0) to identify a true-black optical level. The true-blackoptical level may be used in one or more processes performed by thecontroller (e.g., an auto-exposure process).

In some embodiments, some of the stray-light microlenses 712 maycorrespond to specific color channels (e.g., color channels in imagescaptured using the image sensor 704). For example, if thelight-sensitive pixels 604 are sensitive to one of: red light, bluelight, or green light (e.g., based on an optical filter in thelight-sensitive pixel 604), some of the subsets of light-sensitivepixels 604 may also be sensitive to the same color (e.g., each subset oflight-sensitive pixels 604 may only be sensitive to red, each may onlybe sensitive to blue, or each may only be sensitive to green). Forinstance, one stray-light microlens 712 may be positioned over ninelight-sensitive pixels 604 that are each sensitive to only green lightsignals while a neighboring stray-light microlens 712 may be positionedover nine light-sensitive pixels 604 that are each sensitive to only redlight signals. This may be contrary to the arrangement of coloredlight-sensitive pixels 604 across a bulk of the image sensor 704 (e.g.,nine neighboring light-sensitive pixels may all be sensitive to the samewavelength range, rather than sensitive to different wavelength ranges).For example, the light-sensitive pixels 604 in the bulk of the imagesensor 704 (e.g., the light-sensitive pixels 604 over which stray-lightmicrolenses 712 are not disposed) may be organized in a Bayer filter(i.e., BGRG) arrangement, whereas the light-sensitive pixels 604underneath each of the stray-light microlenses 712 may not be organizedin a Bayer filter arrangement.

In embodiments where one or more subsets of light-sensitive pixels 604underneath corresponding stray-light microlenses 712 may correspond tospecific color channels, a processor (e.g., a camera controller) maydetermine stray light of the corresponding color based on the angles ofincidence of light incident on the respective light-sensitive pixels 604underneath the stray-light microlenses 712. For example, if a subset oflight-sensitive pixels 604 underneath a stray-light microlens 712corresponds to a blue channel, a processor (e.g., a camera controller)may determine an amount of stray blue light incident on the underlyinglight-sensitive pixels 604 based on a determined angle of incidence forthe blue light-sensitive pixels 604 underneath the stray-light microlens712. Such stray light determinations for different color channels may bedone using different subsets of light-sensitive pixels 604 havingdifferent color sensitivities. For example, a first subset oflight-sensitive pixels 604 may correspond to a first color channel ofthe image sensor 704 (e.g., a red color channel), a second subset oflight-sensitive pixels 604 may correspond to a second color channel ofthe image sensor 704 (e.g., a green color channel), and a third subsetof light-sensitive pixels 604 may correspond to a third color channel ofthe image sensor 704 (e.g., a blue color channel). Using the lightsignals detected by the first subset of light-sensitive pixels 604, thesecond subset of light-sensitive pixels 604, and the third subset oflight-sensitive pixels 604, an amount of stray light for a first colorchannel (e.g., red), a second color channel (e.g., blue), and a thirdcolor channel (e.g., green) can be determined. Because the stray-lightdeterminations can be done in a color channel-specific fashion, thestray-light determinations for different color channels can also assistin demosaicing an image captured by the image sensor 704.

Using the stray-light determinations corresponding to multiple subsetsof light-sensitive pixels 604 under corresponding stray-lightmicrolenses 712, a processor (e.g., a camera controller) may makeinferences (e.g., via regression, interpolation, or other techniques)about the stray light incident on other regions of the image sensor 704(e.g., regions where there are not stray-light microlenses 712). Basedon these inferences a plot and/or map of the estimated stray lightacross the image sensor 704 may be generated. Such generated plotsand/or maps may be stored (e.g., within a memory of the camera) and usedin run-time to identify stray light within and/or extract stray lightfrom captured images (e.g., in post-processing a controller may alterthe intensity in certain regions of a captured image based on the storedplot and/or map).

FIG. 8A illustrates an example generated stray-light map. As illustratedin FIG. 8A, the stray-light map may be a plot of stray-light intensitywith respect to pixel location along an x-axis of the image sensor 704.The plot may be generated based on determined stray-light intensitiesfrom one or more subsets of light-sensitive pixels 604 underlying one ormore corresponding stray-light microlenses 712, for example.Additionally or alternatively, in some embodiments, a plot ofstray-light intensity with respect to pixel location along ay-axis maybe generated. The plot illustrated in FIG. 8A may be generated based ona determined angle of incidence for a light signal incident on a firstsubset of light-sensitive pixels 604 on the image sensor 704, adetermined angle of incidence for a light signal incident on a secondsubset of light-sensitive pixels 604 on the image sensor 704, thelocation of the first subset of light-sensitive pixels 604 on the imagesensor 704, and the location of the second subset of light-sensitivepixels 604 on the image sensor 704. Further, in some embodiments,determining the stray-light map (e.g., by a camera controller) mayinclude generating a gamma-encoded histogram or a log-encoded histogramand interpolating values for light-sensitive pixel 604 locations thatare not within the subsets of light-sensitive pixels 604 for which anamount of stray light was determined.

FIG. 8B represents a two-dimensional stray-light map (e.g., atwo-dimensional histogram). As illustrated, a vertical axis of the mapcorresponds to stray-light intensity, and the two horizontal axescorrespond to the horizontal axes of the image sensor 704 (e.g., pixellocation along the y-axis and pixel location along the x-axis). Like theplot illustrated in FIG. 8A, the two-dimensional stray-light mapillustrated in FIG. 8B may be generated based on determined stray-lightintensities from one or more subsets of light-sensitive pixels 604underlying one or more corresponding stray-light microlenses 712. Alsolike the plot illustrated in FIG. 8A, the two-dimensional stray-lightmap illustrated in FIG. 8B may be generated based on a determined angleof incidence for a light signal incident on a first subset oflight-sensitive pixels 604 on the image sensor 704, a determined angleof incidence for a light signal incident on a second subset oflight-sensitive pixels 604 on the image sensor 704, the location of thefirst subset of light-sensitive pixels 604 on the image sensor 704, andthe location of the second subset of light-sensitive pixels 604 on theimage sensor 704. Further, in some embodiments, determining thestray-light map (e.g., by a camera controller) may include generating agamma-encoded histogram or a log-encoded histogram and interpolatingvalues for light-sensitive pixel 604 locations that are not within thesubsets of light-sensitive pixels 604 for which an amount of stray lightwas determined.

As described above, the stray-light maps illustrated in FIGS. 8A and 8Bmay be stored (e.g., within a memory of the camera) and used in run-timeto identify stray light within and/or extract stray light from capturedimages (e.g., in post-processing a controller may alter the intensity incertain regions of a captured image based on the stored plot and/ormap). For example, a controller may be configured to alter payloadimages captured using the image sensor 704 by selectively decreasing anintensity of one or more regions of the payload images based on thedetermined amount of stray light (e.g., based on the generatedstray-light map(s)). Additionally or alternatively, a controller (e.g.,a controller associated with a camera, such as the camera 400illustrated in FIG. 4A or the camera 130 illustrated in FIG. 1) may beconfigured to adjust a pose of a camera, adjust a lens of a camera,and/or apply one or more filters (e.g., neutral-density filters) to acamera or a portion of the camera based on a determined amount of straylight on the image sensor (e.g., based on one or both of the stray-lightmaps illustrated in FIGS. 8A and 8B).

The determined amount of stray light (e.g, the determined amount ofstray-light across the image sensor 704 described by one or morestray-light maps) may be stored in a memory (e.g., a non-transitory,computer-readable medium associated with a camera controller) as alook-up table (e.g., a look-up table that can be accessed by thecontroller in run-time).

In some embodiments, the stray-light maps illustrated in FIGS. 8A and 8Bmay correspond to a single color channel of the corresponding imagesensor 704 (e.g., based on signals detected by light-sensitive pixels604 beneath stray-light microlenses 712 that are of the respective colorchannel, such as nine red-light-sensitive pixels beneath a stray-lightmicrolens 712). In such embodiments, there may be multiple generatedstray-light maps (e.g., one stray-light map for each color channel ofthe image sensor 704). For example, a controller may generate a firststray-light map across the image sensor 704 corresponding to a firstcolor channel, generate a second stray-light map across the image sensor704 corresponding to a second color channel, and generate a thirdstray-light map across the image sensor 704 corresponding to a thirdcolor channel.

FIGS. 8A and 8B are provided by way of example only. It is understoodthat other stray-light maps are also possible and contemplated herein(e.g., stray-light maps with different dimensionality, different units,different resolutions, etc.). In some embodiments, no stray-light mapmay be determined. For example, the raw data associated with signalsdetected from light-sensitive pixels 604 beneath stray-light microlenses712 may be stored (e.g., in a memory, such as a non-transitory,computer-readable medium). Then, when stray-light for a given region ofthe image sensor 704 is to be determined at a later time, a calculationfor that specific region is made based on the stored raw data.

While a plurality of stray-light microlenses 712 are illustrated inFIGS. 7A and 7B and used to produce the stray-light maps illustrated inFIGS. 8A and 8B, FIGS. 7A and 7B are provided by way of example. It isunderstood that in alternate embodiments, other numbers of stray-lightmicrolenses 712 could be included in the image sensor 704 (e.g., agreater number of stray-light microlenses 712 could be included, alesser number of stray-light microlenses 712 could be included, or evenonly a single stray-light microlens 712 could be included). For example,in various embodiments, one, two, three, four, five, six, seven, eight,nine, ten, etc. stray-light microlenses 712 may be included in the imagesensor 704. Additionally or alternatively, while the stray-lightmicrolenses 712 illustrated in FIG. 7A are positioned around the entireperiphery of the image sensor 704, in alternate embodiments, thestray-light microlenses 712 may be positioned in additional oralternative locations on the image sensor. For example, as illustratedin the image sensor 904 of FIGS. 9A and 9B, one or more of thestray-light microlenses 712 may be positioned in the bulk region of theimage sensor 904 (e.g., the region of the image sensor 904 used tocapture light from the surrounding scene). Alternatively, in someembodiments, as illustrated in the image sensor 1004 of FIGS. 10A and10B, the stray-light microlenses 712 may only be positioned in thecorners of the image sensor 1004. Other embodiments are also possibleand are contemplated herein.

III. EXAMPLE PROCESSES

FIG. 11 is a flowchart diagram of a method 1100, according to exampleembodiments.

At block 1102, the method 1100 may include receiving, at a lenspositioned over a first subset of light-sensitive pixels selected from aplurality of light-sensitive pixels that are part of an image sensor, afirst light signal.

At block 1104, the method 1100 may include directing, using the lens,the first light signal toward the first subset of light-sensitivepixels.

At block 1106, the method 1100 may include detecting, by one or morelight-sensitive pixels of the first subset, the first light signal.

At block 1108, the method 1100 may include determining a first angle ofincidence of the detected first light signal.

At block 1110, the method 1100 may include determining, based on thefirst determined angle of incidence, an amount of stray light incidenton the image sensor.

IV. CONCLUSION

The present disclosure is not to be limited in terms of the particularembodiments described in this application, which are intended asillustrations of various aspects. Many modifications and variations canbe made without departing from its spirit and scope, as will be apparentto those skilled in the art. Functionally equivalent methods andapparatuses within the scope of the disclosure, in addition to thoseenumerated herein, will be apparent to those skilled in the art from theforegoing descriptions. Such modifications and variations are intendedto fall within the scope of the appended claims.

The above detailed description describes various features and functionsof the disclosed systems, devices, and methods with reference to theaccompanying figures. In the figures, similar symbols typically identifysimilar components, unless context dictates otherwise. The exampleembodiments described herein and in the figures are not meant to belimiting. Other embodiments can be utilized, and other changes can bemade, without departing from the scope of the subject matter presentedherein. It will be readily understood that the aspects of the presentdisclosure, as generally described herein, and illustrated in thefigures, can be arranged, substituted, combined, separated, and designedin a wide variety of different configurations, all of which areexplicitly contemplated herein.

With respect to any or all of the message flow diagrams, scenarios, andflow charts in the figures and as discussed herein, each step, block,operation, and/or communication can represent a processing ofinformation and/or a transmission of information in accordance withexample embodiments. Alternative embodiments are included within thescope of these example embodiments. In these alternative embodiments,for example, operations described as steps, blocks, transmissions,communications, requests, responses, and/or messages can be executed outof order from that shown or discussed, including substantiallyconcurrently or in reverse order, depending on the functionalityinvolved. Further, more or fewer blocks and/or operations can be usedwith any of the message flow diagrams, scenarios, and flow chartsdiscussed herein, and these message flow diagrams, scenarios, and flowcharts can be combined with one another, in part or in whole.

A step, block, or operation that represents a processing of informationcan correspond to circuitry that can be configured to perform thespecific logical functions of a herein-described method or technique.Alternatively or additionally, a step or block that represents aprocessing of information can correspond to a module, a segment, or aportion of program code (including related data). The program code caninclude one or more instructions executable by a processor forimplementing specific logical operations or actions in the method ortechnique. The program code and/or related data can be stored on anytype of computer-readable medium such as a storage device including RAM,a disk drive, a solid state drive, or another storage medium.

Moreover, a step, block, or operation that represents one or moreinformation transmissions can correspond to information transmissionsbetween software and/or hardware modules in the same physical device.However, other information transmissions can be between software modulesand/or hardware modules in different physical devices.

The particular arrangements shown in the figures should not be viewed aslimiting. It should be understood that other embodiments can includemore or less of each element shown in a given figure. Further, some ofthe illustrated elements can be combined or omitted. Yet further, anexample embodiment can include elements that are not illustrated in thefigures.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopebeing indicated by the following claims.

What is claimed is:
 1. A device comprising: an image sensor comprising aplurality of light-sensitive pixels; a first lens positioned over afirst subset of light-sensitive pixels selected from the plurality oflight-sensitive pixels; and a controller configured to: determine afirst angle of incidence of a first light signal detected by the firstsubset of light-sensitive pixels; and based on the first determinedangle of incidence, determine an amount of stray light incident on theimage sensor.
 2. The device of claim 1, wherein determining the amountof stray light incident on the image sensor comprises comparing thefirst determined angle of incidence to a threshold angle of incidence.3. The device of claim 2, wherein the threshold angle of incidence is10°, and wherein the first determined angle of incidence and thethreshold angle of incidence are measured relative to a vector thatextends perpendicularly from a surface of the image sensor.
 4. Thedevice of claim 1, further comprising a second lens positioned over asecond subset of light-sensitive pixels selected from the plurality oflight-sensitive pixels, wherein the controller is further configured to:determine a second angle of incidence of a second light signal detectedby the second subset of light-sensitive pixels; and based on the seconddetermined angle of incidence, determine the amount of stray lightincident on the image sensor.
 5. The device of claim 4, wherein thecontroller is further configured to generate a stray-light map acrossthe image sensor based on the first determined angle of incidence, thesecond determined angle of incidence, a location of the first subset oflight-sensitive pixels on the image sensor, and a location of the secondsubset of light-sensitive pixels on the image sensor.
 6. The device ofclaim 5, wherein determining the stray-light map comprises generating agamma-encoded histogram or a log-encoded histogram and interpolatingvalues for light-sensitive pixel locations that are not within the firstsubset of light-sensitive pixels or the second subset of light-sensitivepixels.
 7. The device of claim 4, further comprising a third lenspositioned over a third subset of light-sensitive pixels selected fromthe plurality of light-sensitive pixels, wherein the controller is alsoconfigured to: determine a third angle of incidence of a third lightsignal detected by the third subset of light-sensitive pixels; and basedon the third determined angle of incidence, determine the amount ofstray light incident on the image sensor.
 8. The device of claim 7,wherein the first subset of light-sensitive pixels corresponds to afirst color channel of the image sensor, wherein the second subset oflight-sensitive pixels corresponds to a second color channel of theimage sensor, and wherein the third subset of light-sensitive pixelscorresponds to a third color channel of the image sensor.
 9. The deviceof claim 8, wherein the controller is further configured to: generate afirst stray-light map across the image sensor corresponding to the firstcolor channel; generate a second stray-light map across the image sensorcorresponding to the second color channel; and generate a thirdstray-light map across the image sensor corresponding to the third colorchannel.
 10. The device of claim 1, wherein the first subset oflight-sensitive pixels is oriented along a periphery of the imagesensor.
 11. The device of claim 10, wherein the first subset oflight-sensitive pixels is located in a corner of the image sensor. 12.The device of claim 1, wherein the first angle of incidence isdetermined by comparing relative intensities detected by eachlight-sensitive pixel in the first subset of light-sensitive pixels. 13.The device of claim 1, wherein the controller is further configured toalter payload images captured using the image sensor by selectivelydecreasing an intensity of one or more regions of the payload imagesbased on the determined amount of stray light.
 14. The device of claim1, wherein the determined amount of stray light is stored in a memory asa look-up table.
 15. The device of claim 1, wherein the first subset oflight-sensitive pixels comprises a 2×2 array of light-sensitive pixels,a 3×3 array of light-sensitive pixels, or a 4×4 array of light-sensitivepixels.
 16. The device of claim 1, wherein the controller is furtherconfigured to determine a true-black optical level based on thedetermined amount of stray light.
 17. The device of claim 1, wherein theimage sensor is associated with a camera, and wherein the controller isfurther configured to adjust a pose of the camera, adjust a lens of thecamera, or apply one or more filters to the camera based on thedetermined amount of stray light.
 18. The device of claim 1, wherein thedevice is used for object detection and avoidance within an autonomousvehicle.
 19. A method comprising: receiving, at a lens positioned over afirst subset of light-sensitive pixels selected from a plurality oflight-sensitive pixels that are part of an image sensor, a first lightsignal; directing, using the lens, the first light signal toward thefirst subset of light-sensitive pixels; detecting, by one or morelight-sensitive pixels of the first subset, the first light signal;determining a first angle of incidence of the detected first lightsignal; and determining, based on the first determined angle ofincidence, an amount of stray light incident on the image sensor.
 20. Adevice comprising: an image sensor comprising a plurality oflight-sensitive pixels; a plurality of subsets of light-sensitive pixelsselected from the plurality of light-sensitive pixels positioned alongan entire periphery of the image sensor; a plurality of lenses, whereineach lens is positioned over a corresponding subset of light-sensitivepixels; and a controller configured to: determine an angle of incidenceof light detected by each subset of light-sensitive pixels; and based onthe determined angles of incidence, determine a stray-light map acrossthe image sensor.